In order to improve the accuracy of behavior recognition of multi-athletes in football game video, a behavior representation method of multi-athletes for video football game based on scale adaptive local spatial and temporal characteristics was put forward. Behavior recognition was carried on using spatial-timporal interest point to represent behavior of multi-athletes in video football game. Firstly, multi-athletes behavior in the sequence of video football game was regarded as a collection of spatial-timporal interest points in three-dimensional space. Secondly, the set of spatial-temporal interest points was quantified as histogram which has fixed dimension (ie temporal word) by using quantitative technique of histogram. Finally, spatial-temporal codebook was generated by using K-means clustering algorithm. Each spatial-timporal interest point was normalized to ensure its scaling and translation invariance before clustering codebook generated. Experimental results show that the proposed method can greatly reduce the computational amount of the algorithm,and the accuracy of recognition can be significantly improved.